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VA  HEALTH  CARE 

VA  Spends  Millions  on  Post-Traumatic  Stress 
Disorder  Research  and  incorporates  Research 
Outcomes  into  Guidelines  and  Policy  for  Post- 
Traumatic  Stress  Disorder  Services 


Why  GAO  Did  This  Study 

In  addition  to  providing  health  care 
to  veterans,  the  Department  of 
Veterans  Affairs  (VA)  funds  research 
that  focuses  on  health  conditions 
veterans  may  experience.  According 
to  VA,  experts  estimate  that  up  to 
20  percent  of  Operation  Enduring 
Freedom  and  Operation  Iraqi 
Freedom  veterans  have  experienced 
post-traumatic  stress  disorder 
(PTSD)  and  demand  for  PTSD 
treatment  is  increasing.  Because  of 
the  importance  of  research  in 
improving  the  services  that  veterans 
receive,  GAO  was  asked  to  report  on 
VA’s  funding  of  PTSD  research,  and 
its  processes  for  funding  PTSD 
research  proposals,  reviewing  and 
incorporating  research  outcomes  into 
clinical  practice  guidelines  (CPG) — 
tools  that  offer  clinicians 
recommendations  for  clinical 
services  but  do  not  require  clinicians 
to  provide  one  service  over  another — 
and  determining  which  PTSD 
services  are  required  to  be  made 
available  at  VA  facilities.  To  do  this 
work,  GAO  obtained  and  summarized 
VA  data  on  the  funding  of  PTSD 
research  from  its  medical  and 
prosthetic  research  appropriation 
through  its  intramural  research 
program.  GAO  also  reviewed  relevant 
VA  documents,  such  as  those  for 
developing  CPGs  and  those  related  to 
VA’s  2008  Uniform  Mental  Health 
Services  in  VA  Medical  Centers  and 
Clinics  handbook  (Handbook), 
which  defines  certain  mental  health 
services  that  must  be  made  available 
at  VA  facilities.  GAO  also  interviewed 
VA  officials. 


View  GAO-1 1  -32  or  key  components. 
For  more  information,  contact  Randall  B. 
Williamson  at  (202)  51 2-71 1 4  or 
williamsonr@gao.gov. 


What  GAO  Found 

Based  on  VA  data  GAO  obtained  and  summarized,  GAO  found  that  the  amount  of 
funding  VA  provided  for  intramural  PTSD  research  increased  from  $9.9  million  in 
fiscal  year  2005  to  $24.5  million  in  fiscal  year  2009.  From  fiscal  year  2005  through 
fiscal  year  2009,  intramural  PTSD  research  funding  ranged  from  2.5  percent  to 
4.8  percent  of  VA’s  medical  and  prosthetic  research  appropriation.  In  addition,  the 
number  of  PTSD  research  studies  VA  funded  through  the  Merit  Review  Program 
and  the  Cooperative  Studies  Program  (CSP) — VA’s  two  primary  funding 
mechanisms  in  its  intramural  research  program — increased  from  47  in  fiscal  year 
2005  to  96  in  fiscal  year  2009. 

According  to  VA  officials,  intramural  research  proposals,  including  those  on 
PTSD,  are  funded  primarily  according  to  scientific  merit  in  both  the  Merit  Review 
Program  and  CSP.  Proposals  are  evaluated  by  a  panel  of  reviewers  and  scored 
based  on  their  scientific  merit.  Directors  of  VA’s  research  and  development 
services — offices  that  focus  on  different  research  areas  and  administer  VA’s 
intramural  research  program — fund  proposals  based  on  their  scores,  typically  up 
to  a  specified  percentile.  The  number  of  proposals  funded  may  vary  based  on 
budgetary  considerations  and,  for  a  small  number  of  proposals,  responsiveness  to 
VA  research  priority  areas. 

VA  has  a  process  to  review  and  incorporate  relevant  research  outcomes  to 
develop  CPGs  for  a  number  of  topics,  including  PTSD.  VA  relies  on  the  policies  of 
a  joint  VA  and  Department  of  Defense  (DOD)  work  group — comprised  of  VA  and 
DOD  officials — to  ensure  that  systematic  reviews  of  relevant  research  outcomes 
are  conducted  when  issuing  CPGs.  In  brief,  a  systematic  review  is  conducted  to 
identify  the  most  methodologically  rigorous  research  studies  that  are  applicable 
to  each  clinical  question  contained  in  the  CPG.  A  group  of  subject  matter  experts 
then  assesses  the  individual  research  studies  in  order  to  determine  the  overall 
quality  of  evidence  available  for  each  particular  clinical  question,  considers  the 
potential  benefits  and  harms  of  a  clinical  intervention  to  determine  its  net  effect, 
and,  based  on  an  assessment  of  the  overall  quality  of  the  evidence  and  the  net 
effect  of  an  intervention,  develops  recommendations  for  the  CPG. 

According  to  VA  officials,  the  decision  to  require  that  two  PTSD  services — 
cognitive  processing  therapy  and  prolonged  exposure  therapy — be  made  available 
at  VA  facilities  by  including  them  in  the  Handbook  was  based  on  a  review  of 
research  outcomes  and  the  availability  of  existing  resources.  Specifically,  VA 
officials  told  GAO  that  these  two  services  were  strongly  recommended  in  the  2004 
PTSD  CPG  and  had  greater  evidence  supporting  their  effectiveness  than  other 
PTSD  services.  VA  also  told  GAO  that  prior  to  the  Handbook’s  2008  issuance,  VA 
had  already  begun  investing  resources  in  training  programs  for  cognitive 
processing  therapy  in  2006  and  prolonged  exposure  therapy  in  2007.  While  VA 
provided  some  documentation  regarding  the  decision-making  process  for  PTSD 
services,  VA  officials  explained  that  clinical  decision-making  processes  are  not 
typically  expected  to  be  documented  in  a  formal  manner.  VA  officials  told  GAO 
that  they  are  currently  clarifying  language  in  the  Handbook  but  do  not  plan  to 
revise  any  requirements  relating  to  PTSD  services  at  this  time. 

VA  provided  technical  comments  that  GAO  incorporated  as  appropriate. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


January  24,  2011 

The  Honorable  Michael  H.  Michaud 
Ranking  Member 
Subcommittee  on  Health 
Committee  on  Veterans’  Affairs 
House  of  Representatives 

Dear  Mr.  Michaud: 

In  addition  to  providing  health  care  to  about  5  million  veterans  each  year, 
the  Department  of  Veterans  Affairs  (VA)  also  funds  research  that  focuses 
on  the  specific  health  conditions  that  veterans  may  experience.  One 
condition  that  is  examined  in  VA-funded  research  is  post-traumatic  stress 
disorder  (PTSD),  an  anxiety  disorder  that  can  occur  after  a  person  is 
exposed  to  a  life-threatening  event. 1  Veterans  diagnosed  with  PTSD  may 
experience  problems  sleeping,  maintaining  relationships,  and  returning  to 
their  previous  civilian  lives.2  According  to  VA,  experts  estimate  that  up  to 
20  percent  of  Operation  Enduring  Freedom  and  Operation  Iraqi  Freedom 
veterans,3  up  to  10  percent  of  Gulf  War  veterans,  and  up  to  30  percent  of 
Vietnam  War  veterans  have  experienced  PTSD.4  Consequently,  demand  for 
PTSD  treatment  continues  to  grow.  VA  data  show  that  from  fiscal  year 
2004  through  fiscal  year  2008,  the  number  of  unique  veterans  receiving 
treatment  for  PTSD  increased  by  60  percent  from  over  274,000  to  over 
442,000.  In  particular,  the  number  of  Operation  Iraqi  Freedom  and 
Operation  Enduring  Freedom  veterans  who  received  VA  treatment  for 


‘American  Psychiatric  Association,  Diagnostic  and  Statistical  Manual  of  Mental 
Disorders,  Fourth  Edition,  Text  Revision  (Washington,  D.C.,  2000). 

"Those  diagnosed  with  PTSD  may  also  suffer  from  other  ailments,  such  as  depression  and 
substance  abuse. 

3Operation  Enduring  Freedom,  which  began  in  October  2001,  supports  combat  operations 
in  Afghanistan  and  other  locations,  and  Operation  Iraqi  Freedom,  which  began  in  March 
2003,  supports  combat  operations  in  Iraq  and  other  locations.  In  September  2010, 
Operation  Iraqi  Freedom  became  known  as  Operation  New  Dawn. 

4Estimates  for  veterans  who  have  experienced  PTSD  vary.  For  example,  according  to  VA, 
regarding  Operation  Enduring  Freedom  and  Operation  Iraqi  Freedom  veterans,  one  2008 
RAND  study  found  that  approximately  14  percent  of  these  veterans  have  experienced 
PTSD,  while  another  2004  study  by  Hoge  et  at.  estimated  that  from  6  to  20  percent  of  those 
veterans  have  experienced  PTSD. 
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PTSD  increased  from  nearly  4,400  in  fiscal  year  2004  to  over  69,000  in 
fiscal  year  2008. 

Veterans  diagnosed  with  PTSD  can  receive  a  range  of  treatments  to 
manage  their  symptoms,  including  individual  and  group  therapy  and 
medication.  Veterans  can  receive  PTSD  treatment  on  an  outpatient  basis  at 
VA  facilities  such  as  medical  centers,  community-based  outpatient  clinics, 
and  Vet  Centers.5  Veterans  can  also  receive  intensive  treatment  through 
VA  medical  center  inpatient  settings  for  acute  care  needs  and  through 
residential  rehabilitation  treatment  programs  for  more  prolonged 
rehabilitative  care. 

VA  has  guidance  and  policies  in  place  related  to  PTSD  services.  For 
example,  in  2004,  the  joint  VA/Department  of  Defense  (DOD)  Evidence- 
Based  Practice  Work  Group6  issued  a  CPG  for  PTSD.7  The  PTSD  CPG  is  an 
educational  tool  for  clinicians  that  provides  evidence-based 
recommendations8  for  PTSD  services  based  on  a  review  of  PTSD  research 
outcomes.  It  does  not  require  clinicians  to  provide  one  service  over 
another.  However,  in  a  handbook  issued  in  2008 — Uniform  Mental  Health 
Services  in  VA  Medical  Centers  and  Clinics  (Handbook) — VA  for  the  first 
time  required  certain  mental  health  services  to  be  made  available  to 


’Vet  Centers  offer  readjustment  counseling — a  wide  range  of  psychosocial  services  that 
includes  individual  and  group  counseling  and  screening  and  referrals  for  medical  issues — 
to  eligible  veterans  and  their  families. 

6Formed  in  1999  and  composed  of  VA  and  DOD  officials,  the  VA/DOD  Evidence-Based 
Practice  Work  Group  makes  decisions  about  which  clinical  practice  guidelines  (CPG)  for 
specific  conditions  will  be  developed  and  oversees  their  development.  As  of  August  2010, 
the  VA/DOD  Evidence-Based  Practice  Work  Group  had  issued  24  CPGs  on  conditions  such 
as  chronic  heart  failure  and  major  depressive  disorder.  Since  VA  and  DOD  issue  the  CPGs 
in  a  joint  effort,  they  are  intended  to  be  used  by  both  VA  and  DOD  clinicians.  The  VA/DOD 
Evidence-Based  Practice  Work  Group  is  supported  by  VA’s  Office  of  Quality  and 
Performance  and  the  U.S.  Army  Medical  Department’s  Office  of  Quality  Management. 

7The  PTSD  CPG  is  formally  known  as  the  VA/DOD  Clinical  Practice  Guideline  for  the 
Management  of  Post-Traumatic  Stress  (January  2004). 

8Evidence-based  care  refers  to  approaches  that  have  consistently  been  shown  in  controlled 
research  to  be  effective  for  a  particular  condition  or  conditions. 
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veterans  throughout  the  system.9  For  PTSD,  the  Handbook  requires  that 
cognitive  processing  therapy  or  prolonged  exposure  therapy,  two 
evidence-based  psychotherapies,  be  provided  or  made  available  at  VA 
facilities. 10 

Through  its  intramural  research  program,  VA  funds  studies  on  different 
topics,  such  as  PTSD.  VA  intramural  studies  are  conducted  by  VA 
investigators — that  is,  researchers  who  conduct  VA  research  studies.  VA 
intramural  research  refers  to  research  that  is  funded  by  and  conducted 
within  VA.  In  fiscal  year  2009,  VA  received  an  appropriation  of  $510  million 
for  medical  and  prosthetic  research. 11  This  appropriation  funds  VA’s 
intramural  research  program,  including  providing  funding  for  necessary 
equipment  and  supplies. 12  VA  also  receives  three  medical  care 
appropriations  that  support  VA’s  intramural  research  by  paying  some  of 
the  costs  associated  with  this  research,  for  example,  the  salaries  of  VA 


Veterans  Health  Administration  Handbook  1160.01,  Uniform.  Mental  Health  Services  in 
VA  Medical  Centers  and  Clinics  (Sept.  11,  2008).  According  to  the  Handbook,  all  veterans 
with  PTSD  must  have  access  to  cognitive  processing  therapy  or  prolonged  exposure 
therapy.  VA  medical  centers  and  very  large  community-based  outpatient  climes  (serving 
more  than  10,000  unique  veterans  each  year)  must  be  able  to  provide  staff  to  administer 
such  services  at  the  facilities.  Large  (serving  5,000  to  10,000  unique  veterans  each  year)  and 
midsized  (serving  1,500  to  5,000  unique  veterans  each  year)  community-based  outpatient 
clinics  may  provide  these  services  through  telemental  health  or  contract  care  when 
necessary.  In  March  2010,  VA  issued  the  Veterans  Health  Administration  Handbook 
1160.03,  Programs  for  Veterans  with  Post-Traumatic  Stress  Disorder  (PTSD),  to  provide 
additional  information  regarding  the  implementation  of  the  PTSD  requirements  contained 
in  the  2008  Handbook. 

“Psychotherapies  focus  on  changing  individuals’  behaviors,  thoughts,  perceptions,  and 
emotions.  Cognitive  processing  therapy  utilizes  trauma-specific  cognitive  techniques  to 
help  patients  move  past  trauma-related  thoughts  and  progress  toward  recovery.  Prolonged 
exposure  therapy  works  by  helping  individuals  approach  trauma-related  thoughts,  feelings, 
and  situations  that  they  have  been  avoiding  because  of  the  distress  they  cause.  Repeated 
exposure  to  these  thoughts,  feelings,  and  situations  can  assist  with  reducing  the  likelihood 
they  will  cause  distress. 

nVA’s  medical  and  prosthetic  research  appropriation  pays  for  necessary  expenses  in 
carrying  out  programs  of  medical  and  prosthetic  research  and  development  as  authorized 
by  chapter  73  of  title  38,  United  States  Code,  the  authorizing  laws  for  the  Veterans  Health 
Administration.  See  Pub.  L.  No.  110-329,  122  Stat.  3574,  3706  (2008). 

“The  medical  and  prosthetic  research  appropriation  funds  research  on  different  topics.  It  is 
not  limited  to  funding  research  on  PTSD. 


Page  3 


GAO-11-32  Veterans  Affairs  Post-Traumatic  Stress  Disorder  Research 


investigators  who  are  also  VA  clinicians. 13  According  to  VA,  in  fiscal  year 
2009,  VA’s  medical  care  appropriations  totaled  about  $41  billion  and  VA 
provided  $433  million  of  that  amount  to  support  all  research  conducted  at 
VA  facilities. 14 

Because  of  the  growing  demand  for  PTSD  services  and  the  importance  of 
research  in  improving  the  health  care  services  that  veterans  receive,  you 
asked  us  to  report  on  VA’s  funding  of  PTSD  research,  VA’s  processes  for 
reviewing  PTSD  research  proposals,  VA’s  incorporation  of  research 
outcomes  into  clinical  practice,  and  VA’s  process  for  determining  which 
PTSD  services  it  requires  VA  facilities  to  provide  or  make  available.  This 
report  will  describe 

•  how  much  funding  VA  provided  for  intramural  PTSD  research  from  its 
medical  and  prosthetic  research  appropriations  from  fiscal  year  2005 
through  fiscal  year  2009, 

•  how  VA  determines  which  intramural  PTSD  research  studies  will  be 
funded, 

•  how  VA  reviews  PTSD  research  outcomes  and  incorporates  them  into 
its  PTSD  CPG,  and 

•  how  VA  determines  which  PTSD  services  it  requires  VA  facilities  to 
provide  or  make  available. 


To  describe  how  much  funding  VA  provided  for  PTSD  research  from  its 
medical  and  prosthetic  research  appropriations  through  its  intramural 
research  program  from  fiscal  year  2005  through  fiscal  year  2009,  we 
obtained  and  summarized  data  provided  by  VA’s  Office  of  Research  and 


13VA’s  three  medical  care  appropriations  are  (1)  medical  services,  which  provides  funds  for 
the  provision  of  veterans’  health  care  services;  (2)  medical  support  and  compliance,  which 
provides  funds  for  expenses  related  to  the  administration  of  veterans’  health  care  services; 
and  (3)  medical  facilities,  which  provides  funds  for  the  operation  and  maintenance  of  VA’s 
health  care  facilities.  Each  of  these  appropriations  provides  support  to  all  research 
conducted  at  VA  facilities.  For  example,  the  medical  care  appropriations  pay  for  the 
salaries  of  VA  investigators  who  are  VA  clinicians  (according  to  VA,  about  70  percent  of  VA 
investigators  are  VA  clinicians),  administrative  costs  such  as  those  for  payroll  and  human 
resources,  and  logistical  and  infrastructure  costs. 

14VA  could  not  provide  an  estimate  for  the  amount  of  funding  VA  spent  from  the  medical 
care  appropriations  for  intramural  PTSD  research.  Of  the  $433  million,  more  than  half  went 
to  personnel  costs,  including  the  salaries  of  clinicians  and  other  VA  facility  staff. 
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Development  (ORD),  the  office  that  manages  the  Merit  Review  Program 
and  the  Cooperative  Studies  Program  (CSP) — the  two  primary  funding 
mechanisms  within  VA’s  intramural  research  program. 15  VA  officials  define 
PTSD  research  as  “the  effort  to  acquire  generalizable  knowledge  about 
causes,  epidemiology,  susceptibility  and  genetics,  resilience, 
pathophysiology,  prevention  and  treatment  of  post-traumatic  stress 
disorder.”  Based  on  these  data,  for  fiscal  years  2005  through  2009,  we 
calculated  totals  and  trends  for  (1)  overall  intramural  PTSD  research 
funding — including  PTSD  research  studies  and  other  PTSD  research- 
related  funding,  such  as  career  development  awards  provided  to  junior  VA 
investigators  to  conduct  PTSD  studies,  salaries  for  VA  investigators  who 
are  not  VA  clinicians, 16  funding  for  PTSD  research  conducted  within  ORD 
research  centers,  and  PTSD  research  meetings,  and  (2)  the  number  and 
type  of  intramural  PTSD  research  studies  funded.  According  to  VA,  the 
funding  data  we  obtained  from  VA  do  not  include  funds  provided  to 
conduct  or  support  intramural  PTSD  research  from  VA’s  medical  care 
appropriations. 17  Because  of  this,  they  do  not  necessarily  represent  all  the 
funding  VA  provided  for  PTSD  research  during  this  time.  In  addition,  to 
obtain  information  about  how  VA  funds  research  conducted  at  VA,  we 
interviewed  officials  from  VA’s  Veterans  Health  Administration’s  Office  of 
Finance,  ORD,  and  Veterans  Integrated  Service  Networks.18  We  also 
reviewed  VA  documents  regarding  its  total  medical  and  prosthetic 
research  appropriations.  We  tested  the  internal  consistency  and  reliability 
of  the  PTSD  research  data  by  reviewing  the  data  for  obvious  outliers  and 


15The  data  provided  by  ORD  reflect  the  funding  amounts  VA  investigators  were  authorized 
to  use,  not  necessarily  the  amounts  that  were  spent.  VA  officials  said  that  the  authorized 
amounts  are  very  similar  to  the  amounts  that  are  spent. 

“Salaries  for  VA  investigators  who  are  VA  clinicians  are  paid  from  the  medical  services 
appropriation,  one  of  the  medical  care  appropriations. 

“For  example,  VA’s  Veterans  Integrated  Service  Networks,  which  receive  funding  from  the 
medical  care  appropriations  and  include  VA’s  medical  facilities,  may  also  fund  research  by 
VA  investigators,  but  this  research  funding  is  not  considered  part  of  VA’s  intramural 
research  program.  According  to  ORD  officials,  about  95  percent  of  the  PTSD  research 
funded  by  VA  is  funded  by  VA’s  intramural  research  program. 

“The  management  of  VA’s  medical  facilities  is  decentralized  to  21  regional  networks 
referred  to  as  Veterans  Integrated  Service  Networks. 
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performing  consistency  checks.  We  found  the  data  to  be  sufficiently 
reliable  for  the  purposes  of  this  report.19 

To  describe  how  VA  determines  which  intramural  PTSD  research  studies 
will  be  funded,  we  interviewed  VA  officials  responsible  for  managing  VA’s 
intramural  research  program.  We  also  obtained  and  analyzed  VA 
documents  that  describe  VA’s  policies  for  submitting,  reviewing,  and 
funding  intramural  research  proposals.  In  addition,  to  understand  the 
process  VA  uses  to  evaluate  intramural  research  proposals,  we  observed  a 
meeting  in  which  research  proposals  were  evaluated. 

To  describe  how  VA  reviews  PTSD  research  outcomes  and  incorporates 
them  into  its  PTSD  CPG  to  inform  clinical  practice  at  VA,  we  interviewed 
VA  officials  who  are  responsible  for  developing  CPGs.  We  also  obtained 
and  reviewed  the  PTSD  CPG  and  documents  related  to  CPG  development. 

To  describe  how  VA  determines  which  PTSD  services  it  requires  VA 
facilities  to  provide  or  make  available  to  veterans,  we  interviewed  VA 
officials  responsible  for  determining  these  requirements.  We  also  obtained 
and  reviewed  documents  that  describe  the  PTSD  service  requirements. 

We  conducted  this  performance  audit  from  May  2010  through  November 
2010  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 


19In  2010,  the  VA  Office  of  Inspector  General  found  issues  with  ORD’s  validation  of  some 
research  expenditure  data  reported  by  investigators  at  the  VA  Maryland  Health  Care 
System  facility.  See  Department  of  Veterans  Affairs,  Office  of  Inspector  General,  Health 
Care  Inspection:  Inappropriate  Research  and  Development  Data  Entries  Affecting 
Veterans  Equitable  Research  Allocation  (VERA)  Funding  VA  Maryland  Health  Care 
System  Baltimore,  MD  (Washington,  D.C.,  Sept.  23,  2010).  According  to  VA  officials,  the 
PTSD  research  funding  data  we  obtained  for  this  report  were  generated  from  a  different 
data  system  than  the  data  systems  specifically  examined  by  the  VA  Office  of  Inspector 
General.  In  addition,  the  PTSD  research  funding  data  are  entered  by  VA  officials  at  the 
central  office  and  checked  for  accuracy  by  VA  officials  at  both  the  central  office  and  the 
facilities. 
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Background 


VA  manages  its  intramural  research  program  through  ORD.  According  to 
ORD’s  2009  to  2014  strategic  plan,  ORD  has  10  research  priority  areas, 
which  are  topics  of  research  that  are  considered  important  to  VA.20  The 
research  priority  areas  are  the  health  care  needs  of  veterans  who  have 
served  in  Operation  Enduring  Freedom  and  Operation  Iraqi  Freedom, 
aging-related  conditions,  mental  health  care  and  well-being,  chronic 
diseases,  long-term  care  and  caregiving,  deployment-related  exposure  to 
hazardous  environmental  agents,  equity  in  care,  access  in  rural  areas, 
women’s  health,  and  personalized  medicine.  According  to  VA  officials,  all 
of  these  research  priority  areas  could  include  PTSD  research. 

VA  funds  intramural  research  through  the  following: 

VA’s  Merit  Review  Program:  This  program  supports  research  studies 
typically  conducted  by  one  VA  investigator  at  one  VA  facility  and  is 
administered  by  ORD’s  four  research  and  development  services,  each  of 
which  has  a  different  research  focus.  (See  table  1.)  Each  research  and 
development  service  is  responsible  for  soliciting,  reviewing,  selecting,  and 
funding  research  proposals  submitted  to  the  service. 


Table  1 :  The  Office  of  Research  and  Development’s  Research  and  Development  Services 

Research  and  development 
service 

Focus  of  research 

Examples  of  research 

Biomedical  Laboratory 

Biological  or  physiological  principles  in  humans  or 
animals 

Investigations  of  tissues,  blood,  or  other 
biologic  specimens 

Clinical  Science 

Human  beings 

Clinical  studies  examining  interventions  and 
effectiveness;  epidemiological  studies 

Health  Services 

The  interface  of  health  care  systems,  patients,  and 
health  care  outcomes 

Studies  examining  quality,  access,  patient 
outcomes,  and  costs 

Rehabilitation 

Improving  the  quality  of  life  of  impaired  and 
disabled  veterans 

Veteran  rehabilitation  studies;  studies 
focused  on  identifying  technical  solutions  for 
impaired  and  disabled  veterans 

Source:  GAO  analysis  of  Department  of  Veterans  Affairs  information. 


""In  2009,  VA’s  Office  of  Inspector  General  found  that  appropriated  funds  for  VA  research 
addressed  the  broad  spectrum  of  medical  issues  with  which  veterans  contend.  See 
Department  of  Veterans  Affairs,  Office  of  Inspector  General,  Healthcare  Inspection: 
Review  of  the  Veterans  Health  Administration’s  Use  of  Appropriated  Funds  for  Research 
(Washington,  D.C.,  2009). 
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•  VA’s  CSP:  This  program,  which  is  administered  by  Clinical  Science,  funds 
larger-scale,  multisite  clinical  trials  and  epidemiological  research  studies 
on  key  diseases  that  impact  veterans. 

The  Merit  Review  Program  has  research  award  funding  limits,  which  are 
set  by  VA.  In  some  cases,  intramural  research  awards  may  only  be  funded 
for  a  certain  number  of  years.  See  table  2  for  more  information. 


Table  2:  Department  of  Veterans  Affairs  (VA)  Intramural  Research  Study  Funding  Limits  and  Maximum  Length  of  Award  by 
Research  and  Development  Service 


Service 

Funding  limit  per  award  per  fiscal  year 

Maximum  length  of  award  (in  fiscal  years) 

Pilot  study3 

Full  study 

Pilot  study3 

Full  study 

Biomedical  Laboratory 

N/A 

$150,000“ 

N/A 

4 

Clinical  Science 

N/A 

$150,000“ 

N/A 

4 

(5  for  clinical  trials) 

Cooperative  Studies 

N/A 

No  limit 

N/A 

No  limit 

Program 

Health  Services 

$100,000 

$300,000“ 

1 

4 

Rehabilitation 

$75,000 

$300,000 

2 

4 

Source:  GAO  analysis  ofVA  information. 

SA  pilot  study  is  a  study  to  establish  feasibility  or  to  develop  data,  a  technique,  a  concept,  ora 
procedure,  which  is  preliminary  to  undertaking  a  full  study.  Two  of  VA's  Office  of  Research  and 
Development’s  research  and  development  services— Health  Services  and  Rehabilitation— accept 
pilot  study  proposals  in  addition  to  full  study  proposals. 

“According  to  VA  officials,  this  limit  may  be  exceeded  for  funding  the  salaries  of  investigators  who  are 
notVA  clinicians  or  for  equipment. 

‘According  to  VA  officials,  the  total  over  all  years  may  not  exceed  $925,000. 

In  addition  to  individual  studies  conducted  at  VA  facilities,  VA  has  several 
research  centers  and  programs  that  conduct  or  support  PTSD  research. 
For  example,  the  National  Center  for  PTSD  focuses  on  PTSD  research.  VA 
also  has  Research  Enhancement  Award  Programs,  which  help  support 
PTSD  research  by  providing  staff  and  other  resources  to  investigators. 

(For  more  information  on  VA  research  centers  and  research  programs  that 
conduct  or  support  PTSD  research,  see  app.  I.)  According  to  a  VA  official 
from  the  National  Center  for  PTSD,  VA  does  not  fund  most  of  the  PTSD 
research  that  is  being  conducted  today. 


Origin  and  Criteria  for 
Intramural  Research 
Proposals 


Intramural  research  proposals  may  be  service-directed — solicited  by  ORD 
on  specific  topics — or  investigator-initiated —  submitted  by  investigators 
to  ORD  on  their  own  initiative.  Investigators  submit  proposals  either  in 
response  to  a  request  for  proposals  on  a  specific  topic  (for  service- 
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directed  proposals)  or  to  an  open  request  for  proposals  (for  investigator- 
initiated  proposals).  For  both  the  Merit  Review  Program  and  CSP, 
proposals  are  typically  evaluated  in  two  review  cycles  per  year.21 

To  be  considered  for  intramural  research  funding:22 

•  The  proposal  must  be  veteran-centric. 

•  The  proposal  must  have  received  approval  from  the  director  of  the 
medical  center  and  the  research  and  development  office  of  the  medical 
center  where  the  lead  investigator,  known  as  a  principal  investigator,  is 
based.23 

•  The  principal  investigator  and  any  coprincipal  investigators  must 
demonstrate  a  primary  professional  commitment  to  VA,  as  demonstrated 
by  at  least  a  5/8  time  VA  appointment  at  the  time  the  funding  is  awarded 
and  previous  VA  experience,  including  experience  in  research  and  patient 
care.24 

•  Research  must  be  conducted  primarily  on  VA  premises.  The  principal 
investigator  and  any  coprincipal  investigators  must  have  designated 
research  space  within  a  VA  medical  center.25 


“According  to  VA,  proposals  may  be  reviewed  outside  of  a  review  cycle  depending  on 
when  ORD  solicits  requests  for  proposals.  In  rare  cases,  service-directed  proposals  are 
reviewed  by  ad  hoc  reviewers  or  solely  by  research  and  development  service  directors. 

"Each  research  and  development  service  may  have  additional  requirements. 

23The  approval  indicates  that  the  medical  center  has  agreed  to  commit  the  resources,  such 
as  space  and  staff,  necessary  to  conduct  the  research  study. 

“investigators  who  do  not  demonstrate  a  primary  professional  commitment  to  VA  can 
request  a  waiver  of  this  requirement.  According  to  a  VA  document,  a  waiver  may  be  granted 
by  VA  officials  depending  on  the  circumstances  of  the  request,  such  as  for  a  retired 
investigator  who  previously  received  funding  through  the  Merit  Review  Program. 

According  to  VA,  waivers  are  granted  for  about  35  investigators  per  year. 

2BThe  principal  investigator  and  any  coprincipal  investigators  may  also  have  a  designated 
research  space  in  an  approved  non-VA  facility  or  other  VA  facility,  such  as  a  community- 
based  outpatient  clinic. 
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PTSD  Research 
Funding  Increased 
from  Fiscal  Year  2005 
through  Fiscal  Year 
2009 


Overall  intramural  PTSD  research  funding  from  VA’s  medical  and 
prosthetic  research  appropriation  increased  from  $9.9  million  in  fiscal  year 
2005  to  $24.5  million  in  fiscal  year  2009.  The  number  of  intramural  PTSD 
research  studies  funded  through  the  Merit  Review  Program  and  CSP 
increased  from  47  in  fiscal  year  2005  to  96  in  fiscal  year  2009. 


VA  Intramural  PTSD 
Research  Funding 
Increased  from 
$9.9  Million  in  Fiscal  Year 
2005  to  $24.5  Million  in 
Fiscal  Year  2009 


Based  on  the  VA  data  we  obtained  and  summarized,  we  found  that  overall 
intramural  PTSD  research  funding  from  VA’s  medical  and  prosthetic 
research  appropriation  increased  from  about  $9.9  million  in  fiscal  year 
2005  to  about  $24.5  million  in  fiscal  year  2009,  or  by  about  150  percent  (see 
fig.  1).  Overall  intramural  PTSD  research  funding  included  funding  for 
specific  PTSD  studies  as  well  as  for  other  PTSD  research-related  funding, 
such  as  career  development  awards  provided  to  junior  VA  investigators  to 
conduct  PTSD  studies,  salaries  for  VA  investigators  who  are  not  VA 
clinicians,26  funding  for  PTSD  research  conducted  within  ORD  research 
centers,  and  PTSD  research  meetings. 


""Salaries  for  VA  investigators  who  are  VA  clinicians  are  paid  from  the  medical  services 
appropriation,  one  of  the  medical  care  appropriations. 
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Figure  1 :  Department  of  Veterans  Affairs  (VA)  Intramural  Funding  for  Post- 
Traumatic  Stress  Disorder  (PTSD)  Research  Studies  and  Other  PTSD  Research- 
Related  Funding,  Fiscal  Years  2005  through  2009 
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Other  PTSD-related  research  funding 
I  Funding  for  PTSD  research  studies 
Source:  GAO  analysis  ofVA  data. 


Of  the  $80.2  million  provided  for  PTSD  studies  from  fiscal  year  2005 
through  fiscal  year  2009,  $51.3  million,  or  about  64  percent,  was  for  studies 
funded  through  the  Merit  Review  Program.  The  remaining  approximately 
$28.9  million,  or  about  36  percent,  was  for  CSP  studies.  (See  fig.  2.) 
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Figure  2:  Department  of  Veterans  Affairs  (VA)  Intramural  Post-Traumatic  Stress 
Disorder  (PTSD)  Funding  for  Merit  Review  Program  Studies  and  Cooperative 
Studies  Program  (CSP)  Studies,  Fiscal  Years  2005  through  2009 
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Funding  forCSP  studies 
I  Funding  for  Merit  Review  Program  studies 
Source:  GAO  analysis  ofVA  data. 


From  fiscal  year  2005  through  fiscal  year  2009,  intramural  PTSD  research 
funding  ranged  from  2.5  percent  to  4.8  percent  of  VA’s  medical  and 
prosthetic  research  appropriation.  (See  table  3  for  VA  intramural  PTSD 
research  funding  and  VA’s  medical  and  prosthetic  research  appropriations 
from  fiscal  year  2005  through  fiscal  year  2009.)  For  comparison,37 
according  to  a  2009  report  prepared  by  ORD  staff  for  VA’s  National 
Research  Advisory  Council,38  for  fiscal  year  2009,  funding  for  intramural 
traumatic  brain  injury  research  was  about  $14.6  million,  2.9  percent  of  the 
medical  and  prosthetic  research  appropriation.  Funding  for  spinal  cord 


2,Because  of  potential  overlap  in  research  areas,  percentage  estimates  should  not  be  added. 

38The  National  Research  Advisory  Council  is  responsible  for  advising  the  Secretary  of 
Veterans  Affairs  and  the  Under  Secretary  for  Health  on  matters  related  to  VA’s  research. 
The  Office  of  Management  and  Budget  requires  that  the  National  Research  Advisory 
Council  conduct  an  annual  assessment  of  VA’s  intramural  research  program. 
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injury  research  was  $27.2  million,  5.3  percent  of  the  medical  and 
prosthetic  research  appropriation.  Funding  for  intramural  cardiovascular 
disease  and  stroke  research  was  $53.1  million,  10.4  percent  of  the  medical 
and  prosthetic  research  appropriation. 


Table  3:  Department  of  Veterans  Affairs  (VA)  Intramural  Post-Traumatic  Stress 
Disorder  (PTSD)  Research  Funding  and  VA’s  Medical  and  Prosthetic  Research 
Appropriation,  Fiscal  Years  2005  through  2009 

Dollars  in  millions 

Fiscal  year 

VA  intramural  PTSD 
research  funding 

VA’s  medical  and 
prosthetic  research 

appropriation  Percentage 

2005 

$9.9 

$390.2 

2.5 

2006 

13.3 

412.0 

3.2 

2007 

15.7 

446.5 

3.5 

2008 

21.3 

480.0 

4.4 

2009 

24.5 

510.0 

4.8 

Total 

$84.7 

$2,238.7 

3.8 

Source:  GAO  analysis  of  VA  data. 


Note:  VA  intramural  research  is  also  supported  by  VA's  medical  care  appropriations,  which  are  not 
represented  in  this  table.  For  example,  the  medical  care  appropriations  pay  for  the  salaries  of  VA 
investigators  who  are  VA  clinicians  (according  to  VA,  about  70  percentofVA  investigators  are  VA 
clinicians),  administrative  costs  such  as  payroll  and  human  resources,  and  logistical  and 
infrastructure  costs. 


The  Number  of  PTSD 
Studies  Funded  through 
VA’s  Intramural  Research 
Program  Increased  from  47 
in  Fiscal  Year  2005  to  96  in 
Fiscal  Year  2009 


Similarly,  we  found  that  the  number  of  PTSD  studies  funded  from  VA’s 
medical  and  prosthetic  research  appropriations  through  VA’s  intramural 
research  program  increased  from  fiscal  year  2005  through  fiscal  year  2009. 
(See  fig.  3.)  Specifically,  in  fiscal  year  2005,  47  intramural  PTSD  research 
studies  were  funded  while  in  fiscal  year  2009,  96  intramural  PTSD  research 
studies  were  funded.29  This  represented  an  increase  of  more  than 
100  percent.  Of  all  the  studies  funded  each  fiscal  year,  only  a  small  number 
were  CSP  studies. 


29VA  intramural  research  studies  may  be  funded  for  multiple  years  (see  table  2).  Therefore, 
the  same  intramural  PTSD  research  study  funded  in  one  fiscal  year  may  also  be  funded  the 
following  fiscal  year.  In  our  analysis,  this  study  would  be  counted  as  a  funded  study  each 
fiscal  year.  In  addition,  some  studies  were  not  funded  each  fiscal  year  of  their  duration.  For 
example,  a  study  that  was  conducted  from  fiscal  year  2006  through  fiscal  year  2009  may  not 
have  received  funding  in  fiscal  year  2007. 
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Figure  3:  Number  of  Department  of  Veterans  Affairs  (VA)  Intramural  Post-Traumatic 
Stress  Disorder  (PTSD)  Research  Studies  Funded  per  Fiscal  Year,  Fiscal  Years  2005 
through  2009 
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Source:  GAO  analysis  ofVA  data. 


VA  Intramural  PTSD 
Research  Studies  Are 
Funded  Primarily 
According  to 
Scientific  Merit 


According  to  VA  officials,  intramural  research  proposals,  including  those 
on  PTSD,  are  reviewed  and  funded  in  VA’s  Merit  Review  Program  and  VA’s 
CSP  primarily  according  to  scientific  merit. 


VA’s  Merit  Review  Program 

Intramural  research  proposals  submitted  to  VA’s  Merit  Review  Program 
are  reviewed  through  a  series  of  steps  prior  to  funding.  See  figure  4  for  an 
overview  of  the  submission,  review,  and  funding  process  for  proposals 
submitted  to  the  Merit  Review  Program.  (For  more  detailed  information 
on  this  process,  see  app.  II.) 
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Figure  4:  Overview  of  Submission,  Review,  and  Funding  Process  in  Department  of 
Veterans  Affairs’  (VA)  Merit  Review  Program 


Source:  GAO  analysis  of  VA  documents  and  GAO  interviews  with  VA  officials. 


“Proposals  are  submitted  electronically  to  grants.gov,  the  government's  central  grant  identification 
and  proposal  portal.  Proposals  are  then  transferred  to  eRA  Commons,  an  electronic  system  forgrant 
administration  functions,  forVA  processing  and  review. 

First,  investigators  submit  proposals  electronically.  Investigators  typically 
submit  Merit  Review  Program  proposals  to  grants.gov,  the  government’s 
central  grant  identification  and  proposal  portal,30  in  response  to  a  request 
for  proposals.  Submitted  proposals  are  then  transferred  to  eRA  Commons, 
an  electronic  system  for  grant  administration  functions,  for  VA  processing 
and  review.31 


Second,  each  proposal  is  assigned  to  a  merit  review  panel  for  evaluation. 
Each  merit  review  panel  reviews  proposals  in  a  specific  research  topic 
area,  and  is  composed  of  panelists,  typically  associate-level  professors, 
who  are  selected  based  on  their  expertise  in  this  area.32  According  to  VA 
documents,  as  of  2010,  there  were  a  total  of  35  merit  review  panels  across 
VA’s  research  and  development  services.33  The  merit  review  panels 
evaluate  each  proposal  based  on  its  scientific  merit.34  Panelists  consider 


“Depending  on  the  specific  requirements  of  the  request  for  proposal,  a  letter  of  intent  (a 
preliminary  outline  of  a  research  proposal)  may  be  required  prior  to  the  submission  of  a 
research  proposal. 

“Electronic  research  administration,  or  eRA  Commons,  is  managed  and  supported  by  the 
National  Institutes  of  Health  to  provide  support  for  the  funding  of  research  proposals. 

“Merit  review  panelists  may  or  may  not  be  employed  by  VA.  According  to  VA  officials, 
most  merit  review  panelists  have  previous  experience  reviewing  proposals.  According  to  a 
VA  document,  the  number  of  panelists  on  a  merit  review  panel  may  vary.  For  example,  the 
number  of  panelists  on  Rehabilitation’s  merit  review  panels  ranges  from  16  to  69. 

“Biomedical  Laboratory  and  Clinical  Science  share  the  same  merit  review  panels. 

34In  addition,  for  Health  Services,  along  with  scientific  merit,  reviewers  take  into  account 
research  priority  areas  of  Health  Services  when  scoring  proposals. 


Page  15 


GAO-11-32  Veterans  Affairs  Post-Traumatic  Stress  Disorder  Research 


several  criteria  in  evaluating  the  overall  scientific  merit  of  a  proposal.  (See 
table  4  for  criteria  used  to  determine  scientific  merit.) 


Table  4:  Scientific  Merit  Scoring  Criteria  in  the  Department  of  Veterans  Affairs’  (VA) 
Merit  Review  Program 

Criteria 

Description 

Significance 

Supports/advances  the  health  and  health  care  of  veterans  and  research 
field  in  general;  addresses  important  scientific  question/area;  makes  a 
potential  contribution  to  scientific  literature. 

Approach 

Incorporates  current  scientific/theoretical  bases;  is  hypothesis-driven; 
uses  appropriate  research  design/methods  for  addressing  hypothesis; 
ensures  that  feasibility  of  methods  is  clear. 

Innovation 

Addresses  new  concepts,  gaps,  or  both,  in  the  research  area;  addresses 
potential  for  impact  of  findings  on  existing  field  of  research,  treatment 
paradigms,  or  both. 

Environment 

Has  appropriate  knowledge/background  and  resources  (e.g.,  equipment 
and  staff)  to  ensure  completion  of  study. 

Feasibility 

P  rovides  sufficient  evidence  to  determine  that  the  proposed  study  can  be 
successfully  conducted  and  completed. 

Investigator 

Ensures  that  investigators  involved  in  the  proposed  study  are 
appropriately  trained  and  have  expertise  in  the  proposed  area  of 
research. 

Source:  GAO  analysis  ofVA  information. 


Third,  the  merit  review  panelists  score  the  proposals  to  determine  their 
rank.  Each  panelist  provides  a  score  to  each  of  the  proposals  reviewed  by 
the  panel.  The  scores  are  averaged  to  create  a  “priority  score”  for  the 
proposal.36  (See  app.  II  for  specific  scoring  guidelines  given  to  panelists  in 
all  research  and  development  services.)  All  proposals  scored  by  the  merit 
review  panel  are  then  ranked  by  priority  score  among  all  of  the  proposal 
scores  recently  assigned  by  the  merit  review  panel.  The  rank  of  the 
proposal  is  used  to  determine  the  “percentile”  of  the  proposal.36 


3BFor  Health  Services,  prior  to  assigning  a  priority  score  for  a  proposal,  merit  review 
panelists  vote  on  whether  to  approve,  conditionally  approve,  or  reject  the  proposal. 
Conditional  approval  means  the  proposal  is  approved  pending  additions  or  changes.  Only 
approved  and  conditionally  approved  proposals  are  assigned  a  priority  score. 

“According  to  VA,  the  percentile  is  calculated  by  the  proposal’s  rank,  determined  by  its 
priority  score  in  comparison  to  the  scores  of  the  current  and  previous  three  review  cycles, 
minus  0.5,  divided  by  the  total  number  of  applications  being  considered.  For  example,  if  a 
proposal  received  the  fifth  best  priority  score  and  there  were  a  total  of  60  proposals 
reviewed  over  the  four  cycles,  the  percentile  for  that  application  would  be  5  minus 
0.5  divided  by  60,  equaling  .075. 
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Finally,  research  and  development  service  directors  determine  how  many 
proposals  to  fund.  All  of  the  proposals  scored  by  all  merit  review  panels  in 
a  research  and  development  service  in  the  review  cycle  are  ranked 
together  by  their  percentiles  to  be  considered  for  funding.  According  to  VA 
officials,  research  and  development  service  directors  typically  fund  up  to 
the  25th  percentile  of  proposals  in  a  review  cycle,  beginning  with  those 
with  the  most  scientific  merit,  although  the  number  of  proposals  funded 
may  vary  depending  on  the  budget.  According  to  VA,  research  and 
development  service  directors  may  also  choose  to  fund  a  small  number  of 
additional  proposals  at  the  margin  that  respond  to  research  priority  areas. 
For  example,  if  the  fundable  range  determined  by  a  research  and 
development  service  director  was  up  to  the  25th  percentile,  proposals  at 
the  26th  percentile  related  to  research  priority  areas  could  also  be 
considered  for  funding.  37 

YA’s  Cooperative  Studies  Program 

VA  intramural  research  proposals  submitted  to  CSP  are  reviewed  and 
scored  in  a  process  similar  to  that  of  the  Merit  Review  Program  prior  to 
consideration  for  funding.  To  help  develop  the  CSP  proposal,  investigators 
are  assisted  by  members  of  a  CSP  center,38  a  VA  entity  that  provides 
guidance  and  support  for  research  across  multiple  sites.  (See  fig.  5  for  an 
overview  of  the  process  for  submitting,  reviewing,  and  funding  a  CSP 
proposal.  For  more  information  on  ORD’s  CSP  review  process,  see 
app.  III.) 


37 After  a  proposal  is  approved  for  funding  by  the  research  and  development  service 
director,  the  proposal  must  undergo  review  by  an  institutional  review  board,  an 
independent  committee  that  reviews  research  based  on  ethical  considerations.  In  addition, 
depending  on  the  topic  of  the  proposal,  it  may  be  reviewed  by  additional  committees  at  this 
time,  such  as  a  data  monitoring  committee. 

“There  are  nine  CSP  centers:  VA  operates  five  CSP  Coordinating  Centers  to  support 
multisite  clinical  trials  and  four  Epidemiological  Research  and  Information  Centers  to 
support  epidemiology  studies.  In  addition  to  receiving  support  from  a  designated  CSP 
center,  CSP  proposals  that  involve  drugs  or  medical  devices  also  receive  assistance  from 
the  CSP  Clinical  Research  Pharmacy  Coordinating  Center,  and  CSP  proposals  that  involve 
economic  analysis  receive  support  from  VA’s  Health  Economics  Resource  Center. 
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Figure  5:  Overview  of  Submission,  Review,  and  Funding  for  a  Department  of 
Veterans  Affairs’  (VA)  Cooperative  Studies  Program  (CSP)  Proposal 


Source:  GAO  analysis  of  VA  documents  and  GAO  interviews  with  VA  officials. 


“Prior  to  submitting  the  proposal  to  CSP  central  office,  the  principal  investigator  must  have  a  letter  of 
intent  approved  by  the  Director  of  Clinical  Science. 

Before  submitting  a  research  proposal,  investigators  submit  a  letter  of 
intent,  or  a  preliminary  outline  of  a  proposal,  to  the  Director  of  Clinical 
Science  to  be  approved  for  planning  a  CSP  proposal.39  Based  on  the  merit 
of  the  letter  of  intent,  as  determined  by  three  or  more  reviewers,40  the 
Clinical  Science  Director  decides  whether  to  fund  planning  efforts  to 
develop  a  CSP  proposal.41 

When  the  principal  investigator  receives  approval  to  begin  planning 
efforts,  the  Clinical  Science  Director  assigns  a  CSP  center  to  provide 
statistical  and  methodological  guidance  to  the  investigator.  The  director  of 
the  CSP  center  designates  a  project  manager  and  methodologist,  such  as  a 
person  with  expertise  in  biostatistics,  to  provide  guidance  to  the  principal 
investigator.  The  Clinical  Science  Director,  with  recommendations  from 
the  principal  investigator,  then  forms  a  planning  committee  of  additional 
experts  to  assist  in  developing  a  CSP  proposal.  The  planning  committee 
develops  a  CSP  proposal  over  the  course  of  two  planning  meetings. 


Once  a  proposal  is  developed,  the  CSP  center,  on  behalf  of  the  principal 
investigator,  submits  a  hard  copy  proposal  to  the  CSP  central  office  for 


39A  letter  of  intent  typically  contains  an  outline  of  the  proposed  research  study,  including 
justification  for  multiple  sites  where  the  research  will  be  conducted  and  a  list  of  experts 
who  can  assist  in  the  initial  planning  efforts  of  the  research  study. 

"“Reviewers  of  letters  of  intent  are  selected  based  on  their  expertise  in  areas  related  to  the 
topic  of  the  letter  of  intent. 

“Planning  efforts  include  travel  funds  for  personnel  involved  in  developing  the  proposal. 
According  to  VA,  there  are  two  planning  meetings  that  involve  approximately  15  people. 
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evaluation  by  the  Cooperative  Studies  Scientific  Merit  Review  Board.42 
This  board  consists  of  reviewers  who  have  extensive  experience  in  clinical 
research  and  the  conduct  of  clinical  trials  or  epidemiology  studies.43 
Reviewers  evaluate  CSP  proposals  based  on  scientific  merit.  According  to 
VA,  the  scientific  merit  of  a  CSP  proposal  is  defined  by  the  importance  of 
the  proposal,  its  feasibility,  the  clarity  and  achievability  of  its  objectives, 
the  adequacy  of  the  plan  of  investigation,  the  correctness  of  the  technical 
details,  and  the  adequacy  of  the  safeguards  for  the  welfare  of  the 
patients.44  Based  on  these  criteria,  reviewers  discuss  the  general  scientific 
merit  of  the  proposal.  Reviewers  vote  on  whether  to  unconditionally 
approve,  conditionally  approve,  reject  or  defer  with  recommendation  for 
resubmittal,  or  reject  each  proposal.  (See  table  5  for  an  overview  of 
funding  recommendations  provided  by  the  board.) 


Table  5:  Overview  of  Funding  Recommendations  Provided  by  the  Department  of 
Veterans  Affairs’  (VA)  Cooperative  Studies  Scientific  Merit  Review  Board 

Recommendation  for  funding 

Definition 

Unconditional  approval 

The  proposal  is  approved  without  changes  or 
additions  to  the  proposal. 

Conditional  approval 

The  proposal  is  approved  pending  certain 
changes  or  additions  to  the  proposal. 

Rejection  or  defer  with  recommendation 
for  resubmittal 

The  proposal  is  considered  to  be  worthwhile, 
but  is  in  need  of  major  revisions.  A  revised 
proposal  can  be  resubmitted  to  the  Clinical 
Science  Director. 

Rejection 

The  proposal  is  rejected.  The  principal 
investigator  mustsubmita  new  requestfor 
planning  to  submit  a  revised  proposal. 

Source:  GAO  analysis  of  a  VA  document. 


42CSP  proposals  are  currently  not  submitted  electronically;  however,  ORD  officials  estimate 
that  electronic  submission  of  CSP  proposals  through  eRA  Commons  will  begin  in  fall  2011. 

43 According  to  a  VA  document,  members  of  the  Cooperative  Studies  Scientific  Merit  Review 
Board  serve  4-year  terms,  and  ad  hoc  members  can  be  added  depending  on  specific 
expertise  that  may  be  needed  to  review  a  proposal.  As  of  October  2010,  there  were  six 
members  of  the  Cooperative  Studies  Scientific  Merit  Review  Board. 

44CSP  proposals  are  also  reviewed  by  a  human  rights  committee  at  the  designated  CSP 
center  to  determine  if  the  protection  of  patients’  rights  and  welfare  is  adequate.  CSP 
proposals  must  be  approved  by  a  human  rights  committee  prior  to  being  funded. 
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After  the  reviewers  vote,  they  each  provide  scores  for  a  proposal 
recommended  for  funding  based  on  scientific  merit.45  The  scores  are  then 
averaged  to  provide  a  priority  score  for  a  proposal.  Finally,  the  Clinical 
Science  Director  considers  the  priority  scores  of  all  the  proposals  in  that 
review  cycle  and  selects  the  proposals  with  the  strongest  priority  scores 
for  funding.  According  to  VA  officials,  the  number  of  proposals  funded 
may  vary  depending  on  the  budget.46 


VA  Has  Established  a 
Standardized  Process 
to  Review  All 
Relevant  PTSD 
Research  Outcomes 
and  Incorporate  Them 
into  a  PTSD  CPG 


The  VA/DOD  Evidence-Based  Practice  Work  Group,  which  is  responsible 
for  developing  and  updating  all  of  VA’s  CPGs,  has  a  standardized  and 
reproducible  process  to  review  all  relevant  research  outcomes  when 
developing  or  updating  all  CPGs,  including  the  PTSD  CPG.  To  develop  or 
update  a  CPG,  the  VA/DOD  Evidence-Based  Practice  Work  Group 
identifies  and  assigns  a  group  of  VA  and  DOD  clinical  leaders  and  experts 
who  are  knowledgeable  in  the  subject  area  to  work  on  the  CPG.47 
Generally,  the  process  to  develop  or  update  a  CPG  consists  of  the 
following  steps. 

First,  the  assigned  group  of  VA  and  DOD  clinical  leaders  and  experts 
identifies  “clinical  questions”  that  will  be  answered  in  the  CPG.  According 
to  VA  officials,  clinical  questions  can  be  either  broad  or  specific.  For 
example,  the  2004  PTSD  CPG  contained  clinical  questions  regarding 
whether  early  intervention  is  more  effective  than  later  intervention,  and 
whether  certain  interventions,  such  as  different  psychotherapies,  are  more 
effective  than  others. 


•  Second,  in  order  to  minimize  bias,  an  external  contractor  conducts  a 
systematic  review  of  relevant  research  and  selects  and  summarizes  the 
most  methodologically  rigorous  research  studies  that  are  applicable  to 
each  of  the  clinical  questions. 


45Scores  are  given  only  to  conditionally  and  unconditionally  approved  proposals. 

46 After  a  CSP  proposal  has  received  approval  for  funding,  the  proposal  must  undergo 
additional  steps  before  study  subjects  are  enrolled.  For  example,  the  proposal  must  be 
reviewed  by  an  institutional  review  board,  and  an  executive  committee  meets  to  review  the 
operational  and  monitoring  aspects  of  the  study. 

4‘In  addition,  representatives  from  VA’s  Office  of  Patient  Care  Services  are  responsible  for 
identifying  VA  and  DOD  clinical  leaders. 
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•  Third,  after  receiving  summaries  of  the  studies  with  the  highest  level  of 
evidence,  the  VA  and  DOD  group  of  clinical  leaders  and  experts  rates  the 
research  using  an  established  grading  scheme  that  considers 

•  the  level  of  evidence  of  each  research  study — the  scope  and 
methodological  rigor  of  an  individual  study; 

•  the  overall  quality  of  evidence — the  overall  quality  of  all  of  the 
research  that  addresses  a  particular  clinical  question,  considering  the 
level  of  evidence  of  all  the  studies  considered;  and 

•  the  net  effect  of  an  intervention — according  to  the  collective  results  of 
the  studies  considered,  the  intervention’s  benefits  minus  the 
intervention’s  harms. 

•  Finally,  the  assigned  group  of  VA  and  DOD  clinical  leaders  and  experts 
assigns  a  grade  to  each  evidence-based  recommendation  based  on  an 
assessment  of  the  overall  quality  of  evidence  and  the  net  effect  of  the 
intervention.  (See  app.  IV  for  a  detailed  description  of  the  process  used  to 
develop  evidence-based  VA/DOD  CPGs.) 

The  process  for  conducting  a  systematic  review  of  research  outcomes  to 
develop  or  update  a  CPG  is  repeated  as  often  as  is  deemed  necessary  by 
the  VA/DOD  Evidence-Based  Practice  Work  Group  according  to  its  written 
procedures  and  designated  time  frames.48  According  to  VA/DOD  Evidence- 
Based  Practice  Work  Group  documents,  routine  updates  to  the  CPGs 
should  ideally  occur  approximately  every  2  years.  However,  updates  to 
CPGs  often  do  not  occur  every  2  years,  and  VA  officials  told  us  that  some 
CPGs  are  updated  more  frequently  than  others  based  on  availability  of 
resources  and  priority  areas.  Additionally,  VA  officials  reported  that  a  CPG 
will  be  immediately  updated  if  any  evidence-based  recommendation 
contained  in  it  is  identified  as  harmful  to  patients. 

According  to  VA,  the  VA/DOD  Evidence-Based  Practice  Work  Group 
approved  an  update  to  the  2004  PTSD  CPG  on  October  25,  2010,  and 
published  the  update  on  VA’s  Web  site  on  November  17,  2010. 49  According 


48 According  to  VA,  the  development  or  update  of  a  CPG  typically  takes  18  months. 

49 According  to  VA,  the  update  of  the  PTSD  CPG  mainly  strengthens  the  evidence  that  was 
already  contained  in  the  2004  PTSD  CPG.  According  to  a  VA  official,  the  updated  PTSD 
CPG  will  include  a  module  on  complementary  alternative  medicine,  which  was  not 
included  in  the  2004  PTSD  CPG. 
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to  VA  officials,  the  systematic  process  outlined  above  was  used  to  review 
all  relevant  research  outcomes  and  make  evidence-based 
recommendations  for  PTSD  services  to  both  develop  and  update  the  PTSD 
CPG. 


VA  Reviewed 
Research  Outcomes 
and  Used  Existing 
Resources  in 
Determining  Which 
PTSD  Services  to 
Require  Its  Facilities 
to  Provide  or  Make 
Available  to  Veterans 


According  to  VA  officials,  the  decision  to  require  that  cognitive  processing 
therapy  and  prolonged  exposure  therapy  be  made  available  to  veterans 
diagnosed  with  PTSD  at  VA  facilities — as  indicated  in  the  Handbook, 
which  established  certain  requirements  for  mental  health  services  within 
VA — was  based  on  a  review  of  research  outcomes  and  the  availability  of 
existing  resources. 

Review  of  research  outcomes.  According  to  VA,  agency  officials  and 
qualified  subject  matter  experts  reviewed  relevant  research  outcomes  and 
the  quality  of  the  research  to  determine  the  most  efficacious  PTSD 
treatments  available  when  determining  which  PTSD  services  to  include  in 
the  Handbook  and  make  available  to  veterans.  Specifically,  VA  officials 
told  us  that  their  decision  to  include  cognitive  processing  therapy  and 
prolonged  exposure  therapy  in  the  Handbook  was  influenced  by  the  fact 
that  both  of  these  had  been  graded  as  level  “A”  treatments  in  the  2004 
PTSD  CPG  (indicating  that  the  intervention  is  always  indicated  and 
acceptable).50  Furthermore,  VA  officials  said  that  these  two  therapies  had 
greater  evidence  supporting  their  effectiveness  than  other  PTSD  services 
also  graded  as  level  “A”  in  the  2004  PTSD  CPG.  61  In  addition,  VA  officials 
added  that  their  decision  was  validated  by  the  results  of  a  VA- 
commissioned  Institute  of  Medicine  study  published  in  2008  that  reviewed 
the  evidence  for  existing  PTSD  treatments.  According  to  VA,  the  study 
found  that  cognitive  processing  therapy  and  prolonged  exposure  therapy 
were  considered  efficacious  treatments  for  PTSD.52  While  the  Institute  of 


““in  the  CPG,  cognitive  processing  therapy  is  graded  a  level  “A”  treatment  specifically  for 
female  sexual  assault-related  PTSD.  However,  VA  officials  told  us  that  they  also  consider 
cognitive  processing  therapy  to  be  effective  for  PTSD  in  veterans.  See  app.  IV  for  a  detailed 
description  of  the  process  used  to  determine  grades  for  evidence-based  recommendations. 

51Two  other  psychotherapies,  stress  inoculation  training  and  eye  movement  desensitization 
and  reprocessing,  were  also  graded  as  level  “A”  treatments  for  PTSD  in  the  2004  PTSD 
CPG,  but  were  not  included  in  the  Handbook. 

52Institute  of  Medicine,  Treatment  of  Posttraumatic  Stress  Disorder:  An  Assessment  of  the 
Evidence  (Washington,  D.C.,  2008).  The  Institute  of  Medicine  is  an  independent,  nonprofit 
organization  that  works  outside  of  government  to  provide  authoritative  advice  to  decision 
makers  and  the  public.  The  Institute  of  Medicine  found  that  exposure  therapies,  including 
prolonged  exposure  therapy  and  elements  of  cognitive  processing  therapy,  are  efficacious 
treatments  for  PTSD. 
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Medicine  report  was  released  after  VA  had  already  decided  to  include 
cognitive  processing  therapy  and  prolonged  exposure  therapy  in  the 
Handbook,  VA  officials  explained  that  the  Institute  of  Medicine  report  was 
the  basis  for  the  decision  not  to  include  other  PTSD  services  in  the 
Handbook. 

•  Availability  of  existing  resources.  VA  officials  told  us  that  prior  to 
issuing  the  Handbook  in  2008,  VA  had  already  begun  investing 
considerable  resources  to  implement  national  training  programs  for 
cognitive  processing  therapy  and  prolonged  exposure  therapy  in  2006  and 
2007,  respectively.  VA  officials  said  that  they  decided  to  implement  the 
national  training  programs  because  VA  realized  the  need  to  create 
sufficient  capacity  so  that  evidence-based  PTSD  treatments  could  be 
available  to  veterans  throughout  the  VA  system.  VA  explained  that  the 
national  training  programs  were  rolled  out  in  advance  of  the  Handbook’s 
issuance  as  part  of  the  implementation  of  VA’s  Comprehensive  Veterans 
Health  Administration  Strategic  Plan  for  Mental  Health  Services,  which 
called  for  rapid  implementation  of  evidence-based  treatments.  VA  did  this 
to  ensure  that  it  had  the  capacity  to  provide  cognitive  processing  therapy 
and  prolonged  exposure  therapy  to  all  veterans  with  PTSD  for  whom  these 
treatments  were  clinically  appropriate.53  VA  officials  said  that  they  were 
able  to  begin  implementing  national  training  programs  for  cognitive 
processing  therapy  in  2006  and  prolonged  exposure  therapy  in  2007 
because  VA  had  qualified  instructors  to  administer  the  programs  and 
money  available  to  fund  them.  54 


53VA  officials  reported  that  incorporating  cognitive  processing  therapy  and  prolonged 
exposure  therapy  into  the  Handbook  helped  ensure  that  clinicians  who  had  been  trained  in 
these  therapies  prior  to  development  of  the  Handbook  would  be  able  to  offer  and  provide 
them  in  VA  facilities  as  needed.  In  May  2010,  VA’s  Office  of  Inspector  General  reported  that 
as  of  November  17,  2009,  the  total  number  of  mental  health  practitioners  who  had 
completed  either  the  cognitive  processing  therapy  or  the  prolonged  exposure  therapy 
training  was  3,086.  See  Department  of  Veterans  Affairs,  Office  of  Inspector  General, 
Healthcare  Inspection:  Progress  in  Implementing  the  Veterans  Health  Administration’s 
Uniforin  Mental  Health  Services  Handbook  (Washington,  D.C.,  2010). 

“To  help  implement  VA’s  Comprehensive  Veterans  Health  Administration  Strategic  Plan 
for  Mental  Health  Services  (November  2004),  VA  allocated  additional  resources  to  fund 
mental  health  strategic  plan  initiatives  in  fiscal  years  2005  and  2006.  According  to  VA 
officials,  a  part  of  these  additional  resources  was  used  to  pay  for  the  national  cognitive 
processing  therapy  and  prolonged  exposure  therapy  training  programs  until  the  costs  of 
the  programs  could  be  integrated  into  VA’s  congressional  budget  submission. 
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Unlike  the  written  and  standardized  process  that  the  VA/DOD  Evidence- 
Based  Practice  Work  Group  established  to  develop  CPGs,  VA  does  not 
have  a  formal  written  process  or  framework  to  explain  its  decision  for 
including  cognitive  processing  therapy  and  prolonged  exposure  therapy  in 
the  Handbook.55  VA  officials  explained  that  they  followed  a  process  when 
choosing  cognitive  processing  therapy  and  prolonged  exposure  therapy, 
but  added  that  clinical  decision-making  processes  are  not  typically 
expected  to  be  documented  in  a  formal  manner. 

VA  officials  told  us  that  they  plan  to  assess  the  implementation  of  the 
Handbook  and  will  update  PTSD  requirements  in  it  as  needed  or  as  new 
information  or  unexpected  obstacles  arise  in  the  future.56  VA  officials 
stated  that  they  are  currently  clarifying  the  language  regarding  some  of  the 
requirements,  but  do  not  plan  to  revise  any  of  the  requirements  relating  to 
PTSD  services  at  this  time. 


Agency  Comments 


We  provided  a  draft  of  this  report  to  VA  and  received  technical  comments, 
which  we  incorporated  into  our  report  as  appropriate. 


We  are  sending  a  copy  of  this  report  to  the  Secretary  of  Veterans  Affairs. 
The  report  also  is  available  at  no  charge  on  the  GAO  Web  site  at 
http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me 
at  (202)  512-7114  or  williamsonr@gao.gov.  Contact  points  for  our  Offices 


55However,  VA  officials  provided  documentation  regarding  consultation  with  experts  and  a 
list  of  VA  entities  that  participated  in  reviewing  the  Handbook. 

“According  to  VA  officials,  VA  policies  must  be  recertified  every  5  years  to  remain  in  effect. 
The  Handbook  must  be  recertified  on  or  before  September  2013. 
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of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last 
page  of  this  report.  GAO  staff  members  who  made  key  contributions  to 
this  report  are  listed  in  appendix  V. 

Sincerely  yours, 


Randall  B.  Williamson 
Director,  Health  Care 


Enclosures  -  V 
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Appendix  I:  VA  Research  Centers  and 
Programs  That  Conduct  or  Support  PTSD 
Research 


In  addition  to  post-traumatic  stress  disorder  (PTSD)  research  studies  that 
are  conducted  by  individual  Department  of  Veterans  Affairs  (VA) 
investigators,  or  researchers,  VA  also  funds  a  number  of  research  centers 
or  programs  that  conduct  or  support  PTSD  research.  See  table  6  for  a 
description  of  these  VA  research  centers  and  programs. 


Table  6:  Department  of  Veterans  Affairs  (VA)  Research  Centers  and  Programs  That  Conduct  or  Support  Post-Traumatic 

Stress  Disorder  (PTSD)  Research 

VA  research  centers/programs 

Description 

Established 
by  Congress? 

Number  of  centers 
or  programs 
with  a  primary 
focus  on  PTSD 

Office  of  Mental  Health  Services 

The  National  Centerfor  PTSD 

Advances  the  clinical  care  and  social  welfare 
of  veterans  through  research,  education,  and 
training  in  the  science,  diagnosis,  and 
treatment  of  PTSD  and  stress-related 
disorders. 

Yes 

r 

Centers  of  Excellence 

Research  the  causes  and  treatments  of 
mental  disorders  and  use  the  dissemination 
of  education  to  implement  new  knowledge 
into  routine  VA  clinical  practices. 

Yes 

2 

Mental  Illness  Research,  Education,  and 
Clinical  Centers 

Research  the  causes  and  treatments  of 
mental  disorders  and  use  education  to 
implement  knowledge  into  routine  clinical 
practice  in  VA. 

Yes 

4 

Office  of  Research  and  Development 

Centers  of  Excellence 

Create  or  maintain  a  core  program  of 
investigators  to  support  and  facilitate  the 
development  of  research  studies  and  train 
and  mentor  new  investigators. 

No 

8 

Quality  Enhancement  Research  Initiative 
Program 

Enhances  the  quality  and  outcomes  of  VA 
health  care  by  implementing  research 
findings  and  evidence-based 
recommendations  into  routine  clinical 
practice. 

No 

lb 

Research  Enhancement  Award  Programs 

Support  groups  ofVA  investigators  that 
address  specific  medical  problems  of 
veterans  to  promote  innovative  research, 
train  new  medical  research  investigators, 
and  foster  new  research  collaborations 
among  investigators. 

No 

1 

Source:  GAO  analysis  ofVA  information. 


The  National  Centerfor  PTSD,  headquartered  in  White  RiverJ  unction,  Vermont,  has  seven  divisions 
in  five  locations  across  the  country. 
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Appendix  I:  VA  Research  Centers  and 
Programs  That  Conduct  or  Support  PTSD 
Research 


“Although  the  Mental  Health  Quality  Enhancement  Research  Initiative  (one  of  nine  Quality 
Enhancement  Research  Initiative  Centers)  focuses  on  depression  and  schizophrenia,  Office  of 
Research  and  Development  officials  stated  that  within  the  past  few  years  it  has  become  more 
involved  with  PTSD  research. 
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Appendix  II:  The  Review  Process  for  VA’s 
Merit  Review  Program 


Research  proposals  submitted  to  the  Department  of  Veterans  Affairs’  (VA) 
Merit  Review  Program  are  evaluated  in  merit  review  panels  that  each 
review  proposals  in  a  specific  research  topic  area.  Each  merit  review 
panel  is  comprised  of  panelists,  typically  associate-level  professors,  who 
are  selected  based  on  their  expertise  in  the  area.  Panelists  are  responsible 
for  scoring  proposals  based  on  scientific  merit  to  provide  funding 
recommendations.  See  figure  6  for  a  detailed  description  of  the  Merit 
Review  Program’s  process  for  reviewing  research  proposals  and  table  7  for 
the  Merit  Review  Program’s  scoring  guidelines. 
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Appendix  II:  The  Review  Process  for  VA’s 
Merit  Review  Program 


Figure  6:  Summary  of  the  Department  of  Veterans  Affairs’  (VA)  Merit  Review 
Process  for  the  Merit  Review  Program 


Source:  GAO  analysis  of  VA  documents  and  GAO  interviews  with  VA  officials. 


“According  to  VA's  policy,  panelists  who  have  a  conflict  of  interest  with  a  given  proposal  are  required 
to  leave  the  meeting  before  the  proposal  is  discussed  and  do  not  score  the  proposal.  According  to 
VA,  a  conflict  of  interest  exists  when  a  reviewer  has  an  interest  in  a  research  proposal  that  is  likely  to 
bias  his  or  her  evaluation  of  it.  Panelists  are  expected  to  inform  the  portfolio  manager  if  they  have  a 
conflict  of  interest. 

bln  some  instances,  research  proposals  are  streamlined,  meaning  thatthey  are  evaluated  only  by  the 
primary,  secondary,  and  tertiary  panelists,  and  are  not  discussed  by  the  entire  merit  review  panel. 
Proposals  may  be  streamlined  if  the  average  of  the  primary,  secondary,  and  tertiary  panelists' 
preliminary  scores  falls  in  the  upper  half  of  the  range  of  scores,  indicating  less  merit.  According  to  VA 
officials,  proposals  with  a  preliminary  score  of  lower  than  2.2,  indicating  very  good  or  excellent,  are 
also  streamlined  fortwo  research  and  developmentservices.  (See  table  7  for  merit  review  process 
scoring  guidelines.)  In  these  instances,  the  administrator  of  the  merit  review  panel  meeting  asks  the 
panelists  if  anyone  has  an  objection  to  streamlining  the  proposal.  If  a  panelist  objects,  the  proposal  is 
discussed  in  the  merit  review  panel  meeting  and  it  is  not  streamlined.  Streamlined  proposals  with 
preliminary  scores  that  fall  in  the  upper  half  of  the  range  of  scores,  indicating  less  merit,  are  not 
considered  for  funding. 
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Appendix  II:  The  Review  Process  for  VA’s 
Merit  Review  Program 


Table  7:  Scoring  Guidelines  for  the  Department  of  Veterans  Affairs’  (VA)  Merit  Review  Program 


Scoring  range  and  grade 

Description 

1.0  -  1.5 

Excellent 

Proposed  research  addresses  important  scientific  area  that  lacks  needed  knowledge.  Hypothesis(es) 
is  clearly  stated  and  research  design/methodology  is  appropriate.  Research  is  innovative, 
representing  state-of-the-art  science.  Potential  findings  may  have  a  vital  role  in  advancing  the  health 
and  health  care  of  veterans  and  research  in  general.  Resources  listed  suggest  a  very  high  probability 
of  the  study's  completion. 

1.6  -  2.2 

Very  good 

Proposed  research  addresses  important  scientific  area.  Hypothesis(es)  is  clearly  stated  and  the 
research  design/methodology  is  appropriate  with  a  few  minor  exceptions.  Potential  findings  may 
have  an  important  role  to  the  health  and  health  care  of  veterans  and  research  in  general.  Resources 
listed  suggest  a  high  probability  of  the  study's  completion. 

2.3  -  2.8 

Good 

Proposed  research  addresses  a  valid  area  of  investigation.  Hypothesis(es)  is  clearly  stated,  but 
research  design/methodology  contains  key  flaws  that  should  be  corrected.  Potential  findings  may 
contribute  to  the  health  and  health  care  of  veterans  and  research  in  general.  Resources  listed 
suggest  that  the  study  could  be  completed. 

2.9  -  3.4 

Fair 

Proposed  research  requires  further  preliminary  data  to  warrant  investigation  as  a  viable  area  of 
research.  Hypothesis(es)  is  not  clear,  research  design/methods  contain  significant  flaws,  or  both.  It  is 
not  clear  how  potential  findings  would  contribute  to  the  health  and  health  care  of  veterans  and 
research  in  general.  It  is  unclear  whether  the  resources  listed  are  sufficient  to  ensure  study 
completion. 

3.5  -  5.0 

Poor 

Proposed  research  does  not  appear  to  address  an  important  scientific  question/area.  Hypothesis(es) 
is  not  clearly  stated,  research  design/methodology  is  inappropriate  or  contains  uncorrectable  flaws, 
or  both.  Design/methodological  limitations  hinder  any  significant  conclusions  that  would  contribute  to 
the  health  and  health  care  of  veterans,  research  in  general,  or  both.  Resources  listed  do  not  suggest 
thatthe  study  will  be  completed. 

Source:  GAO  analysis  ofVA  documents. 
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Appendix  III:  The  Review  Process  for  VA’s 
Cooperative  Studies  Program 


Research  proposals  submitted  to  the  Department  of  Veterans  Affairs’  (VA) 
Cooperative  Studies  Program  (CSP)  are  reviewed  and  scored  by  the 
Cooperative  Studies  Scientific  Merit  Review  Board.  Reviewers  on  the 
board  are  chosen  based  on  their  expertise  in  clinical  or  epidemiological 
research.  They  typically  serve  4-year  terms,  and  ad  hoc  members  can  be 
added  depending  on  specific  expertise  that  may  be  needed  to  review  a 
proposal.  According  to  VA,  as  of  October  2010,  there  were  six  reviewers 
on  the  board. 

During  the  research  proposal  review  process,  the  study  team — which 
includes  the  lead  researcher  (referred  to  as  the  principal  investigator)  and 
a  methodologist,  such  as  a  person  with  expertise  in  biostatistics — has  an 
interactive  discussion  with  the  board  regarding  the  proposal.1  Reviewers 
evaluate  CSP  proposals  based  on  scientific  merit  and  provide  scores  to 
reflect  their  funding  recommendations.  See  figure  7  for  a  detailed 
description  of  the  review  process  for  CSP  research  proposals. 


‘The  study  team  may  also  include  a  health  economist,  in  cases  where  a  CSP  proposal 
involves  economic  analysis.  In  addition,  the  principal  investigator  may  ask  the  Clinical 
Science  Director  to  allow  other  consultants  to  be  a  part  of  the  study  team,  as  necessary. 
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Appendix  III:  The  Review  Process  for  VA’s 
Cooperative  Studies  Program 


Figure  7:  Summary  of  the  Department  of  Veterans  Affairs’  (VA)  Cooperative  Studies 
Program  (CSP)  Review  Process 


Source:  GAO  analysis  of  VA  documents  and  GAO  interviews  with  VA  officials. 


“According  to  a  VA  document,  the  primary  reviewers  are  typically  a  biostatistician  and  a  clinician. 
Reviewers  are  asked  to  comment  on  the  importance  of  the  proposal,  its  feasibility,  the  clarity  and 
achieva  bility  of  its  objectives,  the  adequacy  of  the  plan  of  investigation,  the  correctness  of  the 
technical  details,  the  adequacy  of  the  safeguards  forthe  welfare  of  the  patients,  and  any  other 
pertinent  features  of  the  proposal.  According  to  VA,  these  criteria  are  components  of  scientific  merit. 

bForCSP  proposals  thatinvolve  economic  analysis,  a  health  economistis  also  present  at  the  board 
meeting  to  discuss  the  proposal.  In  addition,  the  principal  investigator  may  ask  the  Clinical  Science 
Directorto  allow  consultants  to  be  presentatthe  board  meeting  if  review  of  the  proposal  requires 
expertise  in  a  specific  area. 

According  to  VA,  if  there  are  any  economic  issues  raised  relating  to  the  proposal,  the  study  team  will 
be  given  an  additional  5  minutes  to  address  these  issues  with  a  health  economist. 
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Appendix  III:  The  Review  Process  for  VA’s 
Cooperative  Studies  Program 


dAccording  to  a  VA  document,  unconditional  approval  means  a  proposal  is  approved  by  the  board 
without  changes  and  is  recommended  for  funding.  Conditional  approval  means  that  the  proposal  is 
approved  by  the  board  pending  certain  changes  or  additions  to  the  proposal.  Reject  or  defer  with 
recommendation  for  resubmittal  means  the  board  considers  the  proposal  to  be  worthwhile,  but  it  is 
need  of  major  revisions.  In  this  case,  if  the  principal  investigator  chooses  to  submit  a  revised 
proposal,  the  Clinical  Science  Director  may  waive  the  requirementto  submita  letter  of  intent.  If  the 
board  rejects  the  proposal,  the  principal  investigator  must  submit  a  new  letter  of  intent  to  resubmita 
revised  proposal. 

"After  the  board  scores  the  proposal,  a  human  rights  committee  typically  provides  a  general 
assessment  of  the  protection  of  patients'  rights  and  welfare  as  described  in  the  proposal.  For  studies 
with  unique  ethical  considerations,  this  review  may  occur  prior  to  the  review  by  the  Cooperative 
Studies  Scientific  Merit  Review  Board. 
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Appendix  IV:  The  Process  for  Developing 
VA/DOD  Evidence-Based  Clinical  Practice 
Guidelines 


In  1999,  the  Department  of  Veterans  Affairs  (VA)  and  the  Department  of 
Defense  (DOD)  formed  the  VA/DOD  Evidence-Based  Practice  Work  Group 
to  issue  joint  VA/DOD  clinical  practice  guidelines  (CPG) — tools  that 
provide  guidance  and  evidence-based  recommendations  to  clinicians 
regarding  the  most  effective  interventions  and  services  for  a  variety  of 
health  care  topics.1  To  develop  or  update  a  CPG,  the  VA/DOD  Evidence- 
Based  Practice  Work  Group  has  a  standardized  process  to  ensure  that 
systematic  reviews  of  relevant  research  outcomes  are  conducted  in  order 
to  formulate  evidence-based  recommendations  for  prevention, 
assessment,  and  treatment  services. 

To  develop  or  update  a  CPG,  the  VA/DOD  Evidence-Based  Practice  Work 
Group  identifies  two  clinical  leaders — one  from  VA  and  one  from  DOD — 
who  then  help  identify  not  more  than  15  to  20  other  experts  in  the  subject 
area  to  form  a  “guideline  working  group.”2  A  member  of  the  VA/DOD 
Evidence-Based  Practice  Work  Group  is  also  selected  to  be  an  evidence 
chaperone  for  each  CPG  to  ensure  that  conformity  to  prevailing  standards 
for  conducting  high-quality  systematic  reviews  is  upheld.3 

To  determine  the  scope  of  the  CPG,  the  guideline  working  group,  the 
evidence  chaperone,  and  a  facilitator  are  responsible  for  identifying 
clinical  questions  that  are  to  be  answered  by  a  systematic  review  of 
relevant  research  outcomes.4  According  to  VA  officials,  clinical  questions 


Evidence-based  care  refers  to  approaches  that  have  consistently  been  shown  in  controlled 
research  to  be  effective  for  a  particular  condition  or  conditions.  The  VA/DOD  Evidence- 
Based  Practice  Work  Group  makes  decisions  about  which  clinical  practice  guidelines  for 
specific  conditions  will  be  developed  and  oversees  their  development.  As  of  August  2010, 
the  VA/DOD  Evidence-Based  Practice  Work  Group  had  issued  24  CPGs  on  conditions  such 
as  chronic  heart  failure  and  major  depressive  disorder.  Since  VA  and  DOD  issue  the  CPGs 
in  a  joint  effort,  they  are  intended  to  be  used  by  both  VA  and  DOD  clinicians.  The  VA/DOD 
Evidence-Based  Practice  Work  Group  is  supported  by  VA’s  Office  of  Quality  and 
Performance  and  the  U.S.  Army  Medical  Department’s  Office  of  Quality  Management. 

"In  addition,  representatives  from  VA’s  Office  of  Patient  Care  Services  are  responsible  for 
identifying  VA  and  DOD  clinical  leaders. 

3VA  and  DOD  designees  from  the  VA/DOD  Evidence-Based  Practice  Work  Group,  with 
assistance  from  VA’s  Office  of  Patient  Care  Services,  are  responsible  for  selecting  the  VA 
and  DOD  clinical  leaders  and  the  evidence  chaperone.  In  addition,  they  also  select  another 
member  of  the  VA/DOD  Evidence-Based  Practice  Work  Group  to  monitor  the  development 
of  the  CPG. 

4The  facilitator  is  also  responsible  for  keeping  the  guideline  working  group  focused  during 
meetings  and  to  ensure  that  it  focuses  on  the  quality  of  research  outcomes  when 
formulating  evidence-based  recommendations  in  the  CPG. 
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can  be  both  broad  and  specific.  For  example,  the  2004  post-traumatic 
stress  disorder  CPG  contained  clinical  questions  regarding  whether  early 
intervention  is  more  effective  than  later  intervention  and  whether  certain 
interventions,  such  as  different  psychotherapies,  are  more  effective  than 
others.5 

According  to  VA,  in  order  to  answer  these  clinical  questions,  an  external 
evidence  center — an  entity  that  conducts  systematic  reviews  of  research 
on  a  variety  of  topics — is  contracted  to  collect  and  review  all  relevant 
research  (including,  but  not  limited  to,  VA-  and  DOD-sponsored  research) 
to  assess  its  applicability  to  each  clinical  question  under  consideration 
using  explicit  and  reproducible  methods.6  The  evidence  center  then 
focuses  its  review  on  the  best  available  research,  that  is,  high-quality, 
methodologically  rigorous  studies  that  address  health  issues  that  impact 
VA  and  DOD  populations  and  consider  the  effectiveness  as  well  as  the 
harms  and  benefits  of  the  intervention  at  issue.  According  to  VA  officials, 
the  evidence  center  provides  summaries  of  only  the  best  available 
research  to  the  guideline  working  group  for  review.7 

After  receiving  the  summaries,  the  guideline  working  group  reviews  the 
research  in  sequential  steps  using  an  established  rating  scheme  developed 
by  the  U.S.  Preventive  Services  Task  Force  to  formulate  evidence-based 
recommendations.8  See  figure  8  for  an  overview  of  the  steps  that  the 
guideline  working  group  uses  to  formulate  evidence-based 
recommendations. 


“The  post-traumatic  stress  disorder  CPG  is  formally  known  as  The  VA/DOD  Clinical 
Practice  Guideline  for  the  Management  of  Post-Traumatic  Stress  (January  2004). 

According  to  a  VA  official,  an  external  evidence  center  is  used  to  minimize  bias. 

7The  evidence  center  provides  summaries  of  research  of  lower  quality  only  if  no  higher- 
quality  research  is  available. 

sThe  U.S.  Preventive  Services  Task  Force  was  first  convened  by  the  Department  of  Health 
and  Human  Services’  Public  Health  Service  in  1984,  and  since  1998  has  been  sponsored  by 
the  Department  of  Health  and  Human  Services’  Agency  for  Healthcare  Research  and 
Quality.  The  U.S.  Preventive  Services  Task  Force  uses  explicit  criteria  to  grade  the 
scientific  evidence  for  a  broad  range  of  clinical  services  and  develop  evidence-based 
recommendations  for  clinicians  and  health  systems.  While  the  U.S.  Preventive  Services 
Task  Force  updated  its  criteria  in  May  2007,  the  VA/DOD  Evidence-Based  Practice  Work 
Group  currently  uses  U.S.  Preventive  Services  Task  Force  criteria  developed  prior  to  May 
2007. 
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Figure  8:  Overview  of  the  Process  Used  by  the  Guideline  Working  Group 
Established  by  the  Department  of  Veterans  Affairs  (VA)/Department  of  Defense 
(DOD)  Evidence-Based  Practice  Work  Group  to  Grade  Evidence-Based 
Recommendations  for  Clinical  Practice  Guidelines 


Source:  GAO  analysis  of  VA/DOD  documents  and  GAO  interviews  with  VA  officials. 


Level  of  evidence.  First,  the  guideline  working  group  reviews  the 
summaries  to  identify  the  level  of  evidence,  or  the  level  of  methodological 
rigor.  For  example,  research  studies  that  have  the  highest  quality  are 
categorized  as  “I”  (indicating  at  least  one  properly  done  randomized 
controlled  trial),  while  research  studies  of  the  lowest  quality  are 
categorized  as  “III”  (indicating  that  the  research  reflects  the  opinion  of 
respected  authorities,  descriptive  studies,  case  reports,  and  expert 
committees).  (See  table  8.) 


Table  8:  Categorization  of  Ratings  Used  by  the  Guideline  Working  Group 
Established  by  the  Department  of  Veterans  Affairs  (VA)/Department  of  Defense 
(DOD)  Evidence-Based  Practice  Work  Group  to  Identify  the  Level  of  Evidence  of 
Research  Outcomes 


Rating 

Description 

1  At  least  one  properly  done  randomized  controlled  trial 

ll-l 

Well-designed  controlled  trial  without  randomization 

11-2 

Well-designed  cohort  or  case-control  analytic  study,  preferably  from  more  than 
one  source 

11-3 

Multiple  time  series  evidence  with/without  intervention,  dramatic  results  of 
uncontrolled  experiment 

III 

Opinion  of  respected  authorities,  descriptive  studies,  case  reports,  and  expert 
committees 

Source:  VA/DOD  document. 


Overall  quality  of  research.  After  determining  the  level  of  evidence  of 
individual  research  studies,  the  guideline  working  group  makes  a 
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determination  regarding  the  overall  quality  of  all  of  the  research  that 
addresses  a  particular  clinical  question.  The  overall  quality  takes  into 
account  the  number,  quality,  and  size  of  all  of  the  individual  research 
studies  together  as  well  as  the  consistency  of  the  results  between  research 
outcomes  to  determine  the  collective  overall  strength  of  the  research. 
Based  on  this  review,  the  guideline  working  group  determines  the  overall 
quality  of  the  evidence  to  be  good,  fair,  or  poor.9  (See  table  9.) 


Table  9:  Categorization  of  Ratings  Used  by  the  Guideline  Working  Group 
Established  by  the  Department  of  Veterans  Affairs  (VA)/Department  of  Defense 
(DOD)  Evidence-Based  Practice  Work  Group  to  Determine  the  Overall  Quality  of 
Research 


Rating 

Description 

Good 

High-grade  evidence  (1  or  ll-l)  linked  to  health  outcome 

Fair 

High-grade  evidence  (l-or  ll-l)  linked  to  intermediate  outcome 

Or 

Moderate-grade  evidence  (11-2  or  11-3)  directly  linked  to  health  outcome 

Poor 

Level  III  evidence  or  no  linkage  of  evidence  to  health  outcome 

Source:  VA/DOD  document. 

Net  effect  of  the  intervention.  For  interventions  that  were  supported 
by  studies  of  “fair”  or  “good”  overall  quality,  the  guideline  working  group 
evaluates  the  benefits  and  the  potential  harms  to  determine  the  net  effect 
of  the  intervention.10  The  net  effect  of  an  intervention  takes  into  account 
the  benefits  of  the  intervention  minus  the  harms  to  determine  the  overall 
potential  clinical  benefit  that  the  intervention  may  provide  to  patients.  The 
net  effect  of  the  intervention  ranges  from  “substantial”  (meaning  the 
benefit  substantially  outweighs  the  harm)  to  “zero  or  negative”  (meaning  it 
has  no  impact  or  a  negative  impact  on  patients).  (See  table  10.) 


9The  overall  quality  of  the  research  is  considered  “good”  when  the  research  outcomes  are 
consistent  from  a  number  of  higher-level  research  studies  and  have  been  conducted  across 
a  broad  range  of  populations,  thereby  providing  a  high  degree  of  confidence  that  the 
research  outcomes  are  true.  A  “fair”  overall  quality  indicates  that  the  research  outcomes 
could  have  been  a  result  of  true  effects  or  biases  present  across  some  or  all  of  the  research 
studies.  A  “poor”  rating  for  overall  quality  indicates  that  any  conclusion  about  the  research 
outcomes  is  uncertain  because  of  serious  methodological  shortcomings,  sparse  data,  or 
inconsistent  results. 

“The  net  effect  of  the  intervention  of  research  outcomes  that  are  found  to  be  of  poor 
overall  quality  is  not  assessed  because  the  evidence  is  insufficient  to  make  a 
recommendation  for  or  against  routinely  providing  the  intervention. 
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Table  10:  Categorization  of  Ratings  Used  by  the  Guideline  Working  Group  Established  by  the  Department  of  Veterans  Affairs 
(VA)/Department  of  Defense  (DOD)  Evidence-Based  Practice  Work  Group  to  Determine  the  Net  Effect  of  the  Intervention  as 
Described  in  the  Research 

Rating 

Description 

Substantial 

More  than  a  small  relative  impact  on  a  frequent  condition  with  a  substantial  burden  of  suffering 

Or 

A  large  impact  on  an  infrequent  condition  with  a  significant  impact  on  the  individual  patient  level 

Moderate 

A  small  relative  impact  on  a  frequent  condition  with  a  substantial  burden  of  suffering 

Or 

A  moderate  impact  on  an  infrequent  condition  with  a  significant  impact  on  the  individual  patient  level 

Small 

A  negligible  relative  impact  on  a  frequent  condition  with  a  substantial  burden  of  suffering 

Or 

A  small  impact  on  an  infrequent  condition  with  a  significant  impact  on  the  individual  patient  level 

Zero  or  negative 

Negative  impact  on  patients 

Or 

No  relative  impact  on  either  a  frequent  condition  with  a  substantial  burden  of  suffering  or  an  infrequent 
condition  with  a  significant  impact  on  the  individual  patient  level 

Source:  VA/DOD  document. 


Grade  of  evidence-based  recommendation.  In  the  final  step,  the 
guideline  working  group  uses  its  assessment  of  the  overall  quality  of  the 
evidence  and  the  net  effect  of  the  intervention  to  grade  evidence-based 
recommendations.  (See  table  11.) 


Table  1 1 :  Grade  of  Evidence-Based  Recommendation  Assigned  by  the  Guideline  Working  Group  Established  by  the 

Department  of  Veterans  Affairs  (VA)/Department  of  Defense  (DOD)  Evidence-Based  Practice  Work  Group  Based  on  the 

Overall  Quality  of  Evidence  and  the  Net  Effect  of  the  Intervention  as  Described  in  the  Research 

The  net  benefit  of  the  intervention 

Overall  quality  of 
evidence 

Substantial  Moderate  Small  Zero  or  negative 

Good 

A  BCD 

Fair 

B  BCD 

Poor  1  III 

Grade  of 
recommendation 

Description  of  recommendation 

A 

A  strong  recommendation  that  the  intervention  is  always  indicated  and  acceptable 

B 

A  recommendation  that  the  intervention  may  be  useful/effective 

C 

A  recommendation  that  the  intervention  may  be  considered 

D 

A  recommendation  that  a  procedure  may  be  considered  not  useful/effective  or  may  be  harmful 

1  Insufficient  evidence  to  recommend  for  or  against— the  clinician  will  use  clinical  judgment 

Source:  VA/DOD  document. 
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